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Abstract 

Interaction is one of the key variables involved in successful online learning. Previous studies 
analyzing online interaction are limited in their ability to reveal dynamic aspects of interaction in online 
learning environments. The aim of this study is to develop and test the multi-level assessment framework for 
analyzing multi-facets of online interaction. To achieve this purpose, a multi-level assessment framework 
including micro -assessment and macro -assessment was proposed. The multi-level assessment framework, then 
was tested with the data collected from an online course. This study suggests that multi-level assessment 
framework provides us with different kinds of information that helps us to understanding interaction in online 
environments. 



Introduction 

The rapid development, in recent years, of Internet technology has changed the nature of interaction in 
online environments. As a result, there is raised concern over how to assess the interaction of online 
environments. Previous studies analyzing online interaction have focused mainly on either micro -level analysis 
or macro-level analysis. The micro -interaction analysis is to examine the content of the information acquired in 
the process of interaction (Henri, 1991; Offir & Lev, 2000; Oliver & Mcloughlin, 1996). The macro - interaction 
analysis, on the other hand, is to examine the flow or patterns of interaction (Henri, 1991; Levin, Kim, & Riel. 
1990). 

Although these methods provide us with a tool for identifying the nature of the interaction occurring in 
online environments, they are limited in their ability to reveal dynamic aspects of interaction in such 
environments. Each analysis method has a limitation in which it does not completely cover the dynamics of 
online interaction. The multi-facets of interaction in online environments require us to build a more 
comprehensive assessment framework that will encompass various assessment methods proposed in previous 
studies. Analyzing interaction at both micro and macro levels is expected to provide us with different kinds of 
information that helps us to understand interaction in online environments. 

Therefore, the purpose of this article is to propose and test a multi-level assessment framework for 
analyzing interaction in online environments. To achieve this purpose, we first propose the multi-level 
assessment framework for analyzing online interaction. Next, we evaluate the multi-level assessment 
framework with the data collected from an online course. 

Multi-Level Assessment Framework for Analyzing Online Interaction 

An initial study proposed a systemic assessment framework to analyze online interaction. Song (2003) 
proposed a multi-level assessment framework to analyze various aspects of interaction in online environments. 
According to his framework, the unit of analysis at the micro level is the individual message. The content 
analysis of individual messages uncovers the nature of shared information. As a result of this microanalysis, 
each individual message can be divided basically into two main dimensions; cognitive and social. The 
dimensions can again be divided into more specific sub-dimensions according to the nature of the interaction 
between participants. At the macro level, the unit of analysis is multiple messages. Since individual messages 
are related to other messages, they form a multiple message combining those individual messages. Therefore, 
macro -analysis gives a good initial approach to sketching out the big picture of an interaction and can serve as a 
way of identifying messages to be analyzed more deeply. It also gives a context to the individual messages as 
they are analyzed at the micro level. 

However, this framework had a limitation in that it did not provide specific guidelines in its 
application. Therefore, one of the goals of this current study was to investigate whether the framework would be 
useful to analyze online interaction. While testing the framework, we identified critical categories and 
assessment criteria for each category. Tabe 1 shows multi- level assessment framework that includes assessment 
levels, assessment units, assessment categories, and assessment criteria for analyzing online environments. 
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Table. 1 Assessment Levels, Assessment Units, Assessment Categories, and Assessment Criteria 



Level 


Analysis Unit 


Analysis Category 


Analysis Criteria 


Micro 


Individual 

message 


• Cognitive dimension 


• One way communication 

• Process (procedure, expository, explanatory) 

• Cognitive trust 


• Social dimension 


• Two way communication 

• Process (development of identity, 

development of social climate, and managing 
the affective factors) 

• Social trust 


Macro 


Multiple 

messages 


• Patterns of Interaction 


• Work patterns (dropouts, crammers, regular 
periods) 

• Type of participation (lurker, persistent) 


• Depth of interaction 


• Number of threads 

• Numbers of participants per thread 

• Length of posting 


• Purpose of interaction 


• Introduction 

• Clarify, question, elaborate 

• Conclude 



Analyzing interaction at the micro-level, there are two clearly identified dimensions in the literature: 
cognitive and social. We define the cognitive dimension as the processing and organization of information in 
order to create knowledge while the social dimension as social presence including affective elements to create a 
community of learning. In addition to these two broader dimensions, we have identified three categories of sub- 
dimensions that can be used for both cognitive and social interaction research: type of communication, 
processes, and trust. First, in comparing cognitive and social interaction, there is a difference in the way that on- 
line participants communicate. Another subdimension that can be studied in assessing cognitive and social 
dimensions of on-line interaction is the processes used to create knowledge and a community of learning. The 
most commonly used categories for the cognitive dimension are Gunawardena, Lowe, & Anderson’s 
procedural, expository, and explanatory. While Gunawardena, Lowe, & Anderson framework is often used to 
categorize cognitive processes, there is no corresponding framework accepted by experts. In fact, 

Gunawardena, Lowe, & Anderson model does include a category of “social processes’’ in addition to 
procedural, expository, and explanatory. However, we have identified corresponding categories that are the 
elements in creating learning communities: development of identities, developing social climate, and managing 
the affective factors. The last subdimension we have identified is cognitive and social trust. Kanawattanachai 
and Yoo (2002) identify the factors that establish cognitive trust as reduction of complexity, reliability of 
functions and information exchange, and delivery of promised action and information. Social trust is those 
factors such as empathy, emotional support, and free expression, that are the basis for social interaction and 
relationship building. 

The macro analysis looks at long term participation and interaction to identify patterns and trends in 
on-line learning. Based on the literature and our own analysis, we developed three main areas for 
macroanalysis: patterns of interaction, depth of interaction, and purpose of interaction (DeLaat, 2001; Garrison 
& Anderson, 2003; Gunawardena, Lowe, and Anderson, 1997; Henri, 1991; Hwang & Wang, 2004; Mazur, 
2004). While each area of the macro analysis has its own characteristics, it is important to look at how each 
affects the overall pattern of interaction. It is the combination of factors that researchers need to look at when 
determining what is happening at the macro level (Howell-Richardson & Mellar, 1996). 
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The Contexts for Evaluation of Multi-Level Assessment Framework 



Participants 

Multi- level-assessment framework for analyzing interaction in online environments was evaluated with 
the data collected from an online course. The participants in this study were twenty graduate students who 
taking an online course in Educational Technology program at a major university located in the northeastern 
United States. 

Data Collection and Analysis Procedures 

All participants took part in an online group discussion. The online group discussion was conducted as 
an extra-credit activity for the course and was facilitated via a web-based bulletin board, “Web City”. The 
discussion took place over a one week discussion period. Participants were randomly assigned to two different 
types of group discussions: an instructor-moderated discussion or a student-moderated discussion. The 
discussion case involved Internet security issues: “Why privacy and security of personal information on the 
Internet became an increasingly important issue? Who is responsible to protect children's privacy and security? 
What can we do in order to protect children's privacy and security in classroom? ”, Students in a student- 
moderated discussion group discussed and created roles related to the problem case, chose their roles, discussed 
the problem case with their group members, and completed a discussion. On the other hand, the students in the 
instructor-moderated discussion group followed the same procedure, except that they did not create and select a 
role. Data were analyzed using the categories of the multi-level assessment framework developed in this study. 

Results 

Micro-Level Analysis 

Cognitive Dimension 

Communication Type. For the most part, communication in the cognitive dimension is a one-way 
transmittal between the sender and receiver (Chen, Wong, Hsu, 2003). Interaction between teacher and student, 
for example, often falls into the cognitive dimension. As the example below illustrates, the instructor does not 
expect any feedback from the student on the content of the information unless the student has questions. This is 
teacher initiated with the expectation that further interaction will need to come from the instructor. For example, 
in the on-line discussions, the instructor wrote: 

The main task in today (Monday) and tomorrow (Tuesday) is to read the problem case and create 
three important roles that we will play. To create the roles, l hope you find answers to the following 
questions:... [Posts four questions]... Please post a note(s) in which you show a role that you feel most 
important to solve this problem before Tuesday. 

It is clear that this is one-way communication with the instructor initiating the discussion and the 
student expected simply to respond. Chen, Wong, and Hsu (2003) associated this dimension with strictly 
teacher-student interaction. However, student-student interaction can also have this cognitive dimension. In the 
following example, the student, like the instructor above, initiates the interaction and does not expect any 
feedback from the other group members. Any further communication about the information provided will need 
to be initiated by the student that posted the paper. 

The problem is how can we provide for children ’s privacy on the Internet that is effective, cost 
effective, and reliable? The answer is very complex and I am attaching a paper about this. 

Process of Interaction. First, the procedural interaction can take the form of orienting, subdividing the 
problem, and designing (Gay & Lentini, 1995), task description, scheduling, or identifying tools and techniques 
to be used (McFadzean & McKenzie), goal and objective creation (Owen, Pollard, Kilpatrick, and Rumley, 
1998) or teacher designed framework within which students are expected to work (Moaliem, 2003, Thorpe, 
2002). In our research, the instructor gave students a general framework on tasks that needed to be 
accomplished and a schedule of due dates. Throughout the course, at key points when tasks needed to be 
accomplished, he would give further directions on how to accomplish a certain task. Students also set their own 
schedules and procedures within the course such as: 

7 have posted a resonse to ( Student Name ) ’sessay that bridges from the material that I posted 
yesterday. I think that you will find my opinions and views there, as well as in the framing remarks 
that 1 made before posing some of the illunstrations from the AOL model. ..If you have ideas as to how 
to compile our team report, l am open to them. One way to compile a team report might be to thread 
them with one another here. Or perhaps we open another thread to do this. This is new ground that 
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we are exploring so I think that we will have to figure out what we are doing together with this as we 
are doing it. 

Another component to processes is expository interaction. Expository interaction can include 
organization of information, concepts put into different contexts, or representation of knowledge (Cunningham- 
Atkins, Powell, Moore, Hobbs, Sharpe, 2004), evaluation/application, problem identification, definition, 
exploration (Newman, Webb, & Cochrane, 1992), moving from concrete to abstract ideas (McMahen, 1995), or 
summarizing concepts (McFadzean & McKenzie, 2001). An example of this is an instructor posing questions in 
order to illicit student ideas or information such as the sample above when the instructor posed 4 questions for 
the students to answer. Another example of this is a student posting answers to a question. The interaction in 
this component focuses on the content and usually is a result of well-structured instructional design (Moallen, 
2003; Thorpe, 2002). 

Finally, the explanatory interaction, often (although not always) is a response to questions raised 
during expository interaction. Often it is the instructor that will provide explanation, although other students 
can fill this role, especially in smaller group activities. Explanation may require negotiating understanding 
(Gay & Lentini, 1995), integration or general clarification, in-depth clarification, inference, judgment 
(Newman, Webb, & Cochrane, 1992) or determining the information gap and what is needed to close it 
(Kay worth & Leidner, 2001/2002). In the case of our research, the instructor used explanatory interaction to 
clarify tasks and goals in accomplishing tasks. However, he also summarized key points at the end of the tasks, 
which acted to explain the issues and concepts presented in the course. While this is an obvious example of 
explanatory interaction, student explanations also help in the developing understanding. For example, 

I need to offer a disclaimer. The material that I posted to get things going yesterday were elements of 
AOL’s way of responding to these issues. AOL has certainly been very successful as a business in 
general and as one which must be attentive to these privacy and security issues. 

In this case, the student was offering further insight into readings he had posted on the previous day. 

He felt that the student he was writing to may not have understood the basis of the information, based on her 
response to his original posting. 

Cognitive Trust. Cognitive interaction needs information and processes that are reliable, productive, 
and relevant to the task. In the following case, the student is questioning the reliability of the information on 
which she needs to base a paper. She also has identified a flaw in the process, an indication of lack of cognitive 
trust. 

I am not sure if I can write the final paper because I feel there is information missing from all groups. 

I know that there is information missing from the parents group because at least one of { student 
name] ’s and one of my postings wasn’t in the summary. 



Social Dimension 

Communication Type. In developing a learning community, the interaction tends to be two-way 
(Chen, Wong, Hsu, 2003). The receiver is expected to contribute equally to the communication process as the 
sender, initiating feedback even when it is not elicited. The level of communication is deeper and more 
complex, as defined by Henri (1991). Communication initiated by the student to the instructor can be an 
example of the social dimension. The following example demonstrates an equality in the initiation of ideas: 
After posting the responses to a questionnaire : To tell the truth, while l didn 't quite follow your 
schedules, I still think that your initial moderation strategy is pretty impressive and interesting. 
Personally, I think that if this task was not scheduled almost toward the end of this semester, we should 
have been able to proceed this discussion with more fun. Unfortunately, at this point of time, we have 
many homework or projects due, which distracts us from discussion more or less. Isn’t it? 

One limitation to using the direction of communication is that it excludes content and purpose of the 
interaction. In fact, some two-way communication can be cognitive and some one-way communication can 
indicate social interaction. As a result, assessment of this subdimension alone does not indicate one type of 
dimension, but rather it helps identify the nature of the interaction. 

Process of Interaction. The first subdimension, developing identities, focuses on creating a group 
identity. In creating a group identity, members explicitly or implicitly determine the norms of group behavior, 
interaction conventions, shared language, and identification with the community (Kling & Courtright, 2003; 
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McFadzean & McKenzie, 2001; Owen, Pollard, Kilpatrick, & Rumley, 1998). This is achieved through group 
involvement, creating feelings of contentment and belonging, and creating new symbolic convention and 
communication codes(Oren, Mioduser, Nachmias, 2002). In the data we analyzed, group members began their 
e-mails by greeting others with their group name, e.g.“hi government group”. This clearly identified the group 
members with the ‘‘government” group, establishing membership through the team name. Another example of 
establishing identity was the student that referred to “we” in stating the group’s position. By using “we” and 
“us” she is identifying two groups that have different membership requirements. Not everyone can be “us”, 
only those in the business group. Part of the business group, she wrote: 

Would you expect us to comply with those regulations or even contribute to this community’s growth 
once we are out of business? 

The second aspect of social processes, developing social climate, focuses on creating the behavioral 
norms, shared language, and interaction conventions of the group (Kling & Courtright, 2003; Owen, Pollard, 
Kilpatrick, and Rumley, 1998). As a community is created, members need reassurance, trust (Gay & Lentini, 
1995), feedback, advice, and support (McFadzean & McKenzie, 2001). This is often achieved through 
socializing and the exchange of personal information (Moaliem, 2003; Rovai, 2002; Vissar, Plornp, Amirault, 
Kuiper, 2002). This socializing helps to develop group norms and values that are the basis for the on-line 
community. The social climate can be negative or positive. It is obvious that student to student interaction 
makes up a large part of this aspect of social process. For example, one student summarizing another’s ideas 
ends with, “ hopefully I’m representing you correctly! If not, just add at will.” Social climate may need to be 
moderated by the instructor. In fact, instructor interaction may set the tone for the social climate. For example, 
by beginning instructions for student tasks with “I am so happy to work with you”, the instructor is setting the 
tone for social interaction expectations. In other words, all group members should be open to working with 
others in their group. The instructor is also implying that work is a collaborative effort, not a transfer of 
knowledge from instructor to student. He clearly states that he is working with the students, not that the 
students are doing work for the instructor. 

The last aspect of creating a community is recognizing and managing the affective factors that 
students, the group, and even the instructor have. This may include participant emotions and feelings (Vissar, 
Plomp, Amirault, & Kuiper, 2002), motivation, conflict and conflict resolution, student control and efficacy 
(Moaliem, 2003), tolerance of ambiguity, and comfort with technology. This is the hardest subdimension to 
identify, since affect is not always expressed explicitly. In face-to-face interactions, participants can see non- 
verbal social cues such as body language, eye contact, and changes in the voice. On-line interaction does have 
non-explicit social cues, however. For example, the student in the following posting: 

Many of the teacher concerns are similar to the parental concerns, because the teacher can give online 
consent for the parent in a school setting. 

Social Trust. Social trust is based on empathy, free expression and emotional bonds (Kanawattanachai 
& Yoo, 2002). These are the affective attributes that are the basis for community and the creation of identity 
within the community. Social trust can affect group processes (Hiltz, 1998), group loyalty, motivation, and 
shared knowledge creation. One major cause for lack of social trust is a lack of social presence by both 
instructor and/or students. The following is an example of how social trust is established: 

I leave for Kansas City at 6:25 cun. Thursday, arrive back in Philly at 11:45 p.m., then take my dad to 
Hopkins for an appointment in Baltimore at 8:30 a.m. Firday.,.1 will not be able to respond to 
everyone ’s posts tomorrow, in other words. I will check in and post on Friday evening. 

In this case, the student is explaining his lack of social presence for the next few days. As a result, he is 
creating expectations, which establishes social trust. He also gives some personal information which can create 
feelings of empathy and an emotion bond with the group members. 

Macro -Level Analysis 

Patterns of Interaction 

Looking at the posting patterns, who posted when, we could get an idea of student work patterns. As 
shown in the Table 2, we found that one group did not post during last three days. Although more than one 
week’s worth of data would be necessary to determine trends, looking at postings throughout the entire course, 
we could determine if this was their regular work pattern. According to Hwang & Wang (2004), there are three 
different types of student work patterns: dropouts, crammers, and regular periodic reading learners. Dropouts 
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tend to have bursts of activity at the beginning of an on-line activity, then stop contributing and/or drop out of 
the course. Crammers have bursts of posting activity around due dates or at the end of an activity, not 
contributing until they have to. Regular periodic reading learners may post regularly or may have multiple 
bursts of activity. However, they develop a sense of presence throughout the activity. By identifying student 
work patterns, researchers can test various instructional designs for effectiveness, identify factors that enhance 
or impede participation, and determine system usage rates. As shown in the Table below, students in group A 
seems to be more dropouts where as those in group B seems to be more regular periodic learners. 

Table 2. Patterns of interaction 

















Date 














Total 


Person 


3/24 


3/25 


3/26 


3/27 


3/28 


3/29 


3/30 


3/31 


4/1 


4/2 


4/3 


4/4 


4/5 




Group A 






























1 


6 




2 


8 




3 


4 














23 


2 






1 








1 














2 


3 




1 


1 


2 


1 




2 














7 


4 




1 






1 






2 












4 


5 




1 


1 




1 




1 




1 










5 


6 






1 










2 












3 


7 






2 


2 


1 


1 




1 


1 








1 


9 


8 






2 






















2 


9 






2 


1 










1 










4 


Total 


6 


3 


12 


13 


4 


4 


8 


5 


3 








1 


59 


Group B 






























1 


5 


4 


4 


3 




7 


1 






2 








26 


10 




2 


1 


1 










1 


1 








6 


12 


















2 










2 


13 


1 




3 


1 


1 


















6 


14 




3 




1 


1 


















5 


15 




2 


1 


1 


2 












1 






7 


16 




1 


2 


8 


3 








3 










17 


17 






1 




2 


















3 


18 








1 




1 
















2 


Total 


6 


12 


12 


16 


9 


8 


1 




6 


3 


1 
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Related to the patterns of interaction is the type of participant in the interaction process. In the macro - 
level analysis, we are concerned with how a participant affects the interaction process, not the prescribed role 
he/she or the group assigns the participants. Mazur (2004) identifies two of these roles as the Lurker and the 
Persistent. The Lurkers may not make their presence known except at key points in the course (such as 
assignment due dates). However, their postings indicate that they are keeping up with the group interaction. 
The Persistent contributes despite negative or no feedback. In table 1, Student # 9 was a Lurker. There was no 
posting from March 27 to April 1. However, in reviewing the April 1 posting, many of the group’s postings 
were incorporated into the student’s final comment. This indicates that the student, while not actively posting, 
did review other group members’ contributions. 

Depth of Interaction 

The example in Figure 1 demonstrates how multiple threads can be related. Message 617 resulted in 5 
direct postings. However, two additional threads were indirectly related to this initial message. One was a 
group discussion without instructor direction generated as a result of message 617; the other was a summary of 
discussion (appointment of group roles) as a result of message 617. The breadth of discussion in this case was 
much more than an index of threads would indicate. The complexity of the interaction and the non-linear 
sequencing are more accurately illustrated in the conversation map. Looking at the lengths of the postings and 
the time between postings help indicate the level of urgency in the interaction and the amount of thought that 
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may have gone on in posting. For example, in message 639 the student indicates that she is confused about the 
assignment. The message is short (less than 10 lines) indicating urgency. Three additional messages are 
exchanged in a direct thread within the next 24 hours. Each of these are medium or short messages. Comparing 
this with the related thread starting with message 660, these messages are longer and posted with more time 
between each one. The depth of the interaction appears to be deeper with message 660 than 639, even though 
both have the same number of postings. On the other hand, message 639 appears to be more urgent, with a need 
for a definite conclusion. 

Purpose of Interaction 

In order to understand what is going on in a sequence, each posting should be characterized as to what 
it is doing in the interaction. Mazur (2004) identifies three different characterizations of a posting: initiation, 
repair, and close. She further defines repair as including clarification and elaboration. We would add to this 
questioning. In initiation, the topic thread is introduced. Message 617, for example, begins the discussion by 
giving instructions on what the assignment is and how students are expected to proceed in the discussion. In 
response to this, some students elaborate by posting their answers to the assignment (messages 658, 620, and 
626). By posting the assignment, each of the students also finishes the conversation/topic. However, message 
655 elaborates by postponing the assignment. As a result, the topic is picked up later under another thread 
starting with message 660. Clarification is usually in response to a question or problem. In message 639, for 
example, the student is unsure of the assignment (resulting in a question). The instructor clarifies the 
assignment. The student elaborates on the assignment, indicating why there was confusion. The instructor then 
clarifies the source of the confusion and finishes the topic. In reviewing the purpose of the interaction, the end 
of the threads should have a concluding statement. However, when a conclusion is missing, this is an indication 
that either the topic is being continued somewhere else (as with message 655) or it has not been thoroughly 
discussed. By analyzing the sequencing of the conversations, researchers can determine how best to facilitate 
on-line interaction, interaction motivation, barriers to interaction, and timing of assignments and modules. This 
type of analysis also allows for the identification of related threads and ideas between threads (Henri, 1991; 
Howell -Richardson & Mellar, H., 1996). 



Conclusion 

The purpose of this paper was to develop and test the multi-assessment framework for analyzing the 
multi-facets of online interaction. The macro analysis helps researchers to identify trends and generate questions 
that can only be answered in a microanalysis. On the other hand, the microanalysis needs a context outside of 
the message that only a macro analysis can provide. 

The application of this framework has provided several areas for further studies. First, in applying the 
microanalysis to an actual on-line discussbn, we discovered that it was difficult in some group interactions to 
separate the cognitive and social interaction. Henri (1991), in fact, developed four categories, social, cognitive, 
metacognitive, and interaction. Since we are looking at all interaction, the last category could be renamed socio- 
cognitive interaction. Future research should look at those aspects of interaction, especially within group 
processes, where there is both social and cognitive interaction at the same time. Second, The macro analysis of 
on-line interaction must include various factors as: technology used, the interaction environment (synchronous 
vs. asynchronous), stated goals of the interaction (topic, field of study, task), and events or unplanned 
interruptions that took place during the interaction (Garrison & Anderson, 2003; Mazur, 2004). This description 
will explain any constraints or limitations in the macro-level analysis. Finally, this framework needs to be 
tested in various contexts and needs to establish interrater reliability. 
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Note 1. Response Code: I=Initiate E=Elaborate Q=Question C=Clarify F=Finish N=Not related . Note 2. Length Code: S<10 lines M: 10-20 lines L>20 lines 
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